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1.  Introduction 


Present-day  Soldiers  receive  an  unprecedented  information  flow  from  numerous  sources  in  order 
to  execute  network-centric  warfare.  Such  information  overload  can  result  in  extremely  high  cog¬ 
nitive  workload,  which  can  subsequently  reduce  situational  awareness  and  lower  the  quality  of 
performance.  This  problem  can  be  conceptualized  as  a  multi-tasking  issue  or  merely  as  a  process¬ 
ing  limitation.  Consequently,  information  providers  are  challenged  with  identifying  ways  to 
disseminate  and  convey  information  as  efficiently  as  possible  so  that  a  Soldier’s  comprehension 
and  decision  making  are  optimized,  while  overload  and  distraction  are  minimized. 

A  promising  conceptual  framework  for  the  information  overload  problem  is  Wickens’  (2002) 
Multiple  Resource  Theory  (MRT).  The  premise  of  MRT  is  that  different  resources  exist  for 
processing  different  modalities  of  input  (i.e.,  different  cognitive  resources  exist  for  processing 
visual,  audio,  and  tactile  information).  Therefore,  when  information  is  presented  from  different 
modalities,  the  perfonnance  decrement  attributable  to  infonnation  overload  should  be  smaller  than 
when  information  is  presented  from  a  single  modality  to  communicate  the  same  amount  of  infor¬ 
mation.  Simply  stated,  MRT  proposes  that  (a)  people  have  several  independent  capacities  with 
resource  properties;  (b)  some  resources  can  be  more  easily  used  in  tandem,  while  other  combina¬ 
tions  are  more  difficult  and  would  be  performed  more  sequentially;  (c)  tasks  using  compatible 
resources  can  usually  be  performed  together;  and  (d)  competition  for  the  same  sensory  modality 
can  produce  interference.  MRT  explicates  these  capacities  and  contingencies  (Wickens,  2002). 

As  an  example,  until  recently,  Soldiers  received  information  primarily  through  visual  presentation 
(map,  compass,  computer  screen).  When  a  large  amount  of  information  is  presented  solely  through 
the  visual  modality,  cognitive  overload  becomes  problematic  and  can  have  negative  effects  on 
performance.  MRT  advises  reducing  the  volume  of  information  presented  in  one  mode  (visually) 
by  offloading  to  one  or  more  other  modes  (e.g.,  audio,  tactile  modes). 

Numerous  researchers  have  approached  the  information  overload  problem  with  MRT  as  the 
theoretical  rationale  (explicitly  or  implicitly)  for  using  multiple  modes  of  presentation  to  reduce 
workload  (Wickens,  2002).  In  order  to  benefit  from  the  vast  amount  of  research  on  this  topic,  the 
research  needs  to  be  accumulated  and  classified  on  multiple  dimensions  such  as  the  type  of  task  and 
the  demands  associated  with  their  use.  Organization  of  the  multimodal  literature  is  warranted  to 
support  the  application  of  research  findings  to  display  design.  There  is  interest  in  detennining  the 
extent  to  which  type  of  display  modality  can  explain  variance  in  performance  and  impact  the  use  of 
cognitive  resources. 

The  purpose  of  this  research  was  to  develop  a  framework  to  enable  classification,  evaluation,  and 
comparison  of  multimodal  display  research  based  on  task  demands,  display  characteristics,  research 
design  and  individual  differences.  In  this  report,  we  describe  the  process  by  which  a  bibliographic 
database  was  developed  and  organized.  First  the  framework  was  specified,  which  then  guided  the 
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identification  and  review  of  research  and  theory-based  articles  that  were  included  in  the 
bibliography. 

The  purpose  of  this  research  was  to  develop  a  framework  to  enable  classification,  evaluation, 
and  comparison  of  multimodal  display  research  based  on  task  demands,  display  characteristics, 
research  design,  and  individual  differences.  In  this  report,  we  describe  the  process  by  which  a 
bibliographic  database  was  developed  and  organized.  First  the  framework  was  specified,  which 
then  guided  the  identification  and  review  of  research  and  theory-based  articles  that  were  included 
in  the  bibliography.  This  effort  then  led  to  meta-analytical  comparisons  of  effect  size  and  direction 
across  empirical  investigations  (Burke  et  ah,  2006). 


2.  Method 


Elliott  and  Redden  (personal  communication,  2005)  developed  a  framework  to  organize  investiga¬ 
tion  and  research  experiments.  This  initial  framework  was  refined  and  augmented  by  the  research 
group  at  University  of  South  Florida  (Coovert,  Gray,  Elliott,  Redden,  2006).  The  resulting  multi¬ 
modal  framework  represents  the  culmination  of  an  extensive  dialogue  between  the  multimodal 
literature  and  the  preliminary  framework.  The  following  sections  describe  the  development  of  the 
multimodal  framework  and  development  of  the  multimodal  database. 

2,1  Development  of  the  Framework 

The  multimodal  framework  (2005)  (see  figure  1)  was  a  preliminary  step  to  understanding  the 
research  on  multimodal  information  processing.  The  use  of  the  initial  framework  in  reviewing 
research  articles  required  deliberate  consideration  and  consistent  description  of  several  study  charac¬ 
teristics.  The  first  step  was  to  examine  each  of  the  initial  framework  components  for  inclusion  in 
the  multimodal  framework  (see  figure  1  for  original  components).  It  was  determined  that  the  frame¬ 
work  could  benefit  from  further  development  to  make  it  more  comprehensive  (new  components  to 
address  research  design,  theory,  results,  conclusions,  and  coding  of  variables  for  meta-analytic 
purposes),  as  well  as  from  modifications  of  the  existing  components  to  improve  clarity  and  focus. 

The  revised  multimodal  framework  was  developed  to  accommodate  additional  components  in  the 
form  of  the  article  tracking  sheet  (see  figure  2). 

Since  the  multimodal  framework  was  intended  to  guide  the  review  of  research  literature,  we  decided 
that  the  format  should  be  conducive  to  the  review  of  an  article.  The  initial  framework  (see  figure  1) 
existed  as  an  electronic  spreadsheet  (columns  were  com-ponents,  rows  were  research  studies),  where¬ 
as  the  article  tracking  sheet  took  the  form  of  a  five-page  Word1  document,  designed  to  accompany  a 
research  article.  The  article  tracking  sheet  encompassed  some  of  the  initial,  several  modified,  and 
some  new  framework  component  fields,  which  were  based  on  a  preliminary  literature  review. 


'Word  is  a  trademark  of  Microsoft  Corporation. 
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Figure  2.  Article  tracking  sheet. 
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2.2  Literature  Review 

In  an  effort  to  locate  articles  spanning  a  variety  of  research  domains,  databases  employed  for  the 
literature  search  included  ACM  (Association  for  Computing  Machinery;  http://www.acm.org/), 
IEEE  (Institute  of  Electrical  and  Electronics  Engineers;  http://www.ieee.org/portal/site),  Psyclnfo 
(http://www.apa.org/psycinfo/),  Web  of  Science  (http://scientific.thomson.com/products/wos/), 
DTIC  (Defense  Technical  Information  Center;  http://www.dtic.mil/),  and  CSA  (http://www.csa. 
com/).  A  number  of  keywords  were  applied  to  each  database  to  define  the  searches  (e.g.,  multi¬ 
modal,  visual,  tactile,  audio,  haptic,  interface,  display,  dual  task,  multitask).  The  abstracts  of 
articles  returned  by  the  databases  were  examined,  and  approximately  900  references  that  were 
initially  deemed  relevant  were  loaded  into  an  on-line  bibliographic  management  software  (www. 
Refworks.com)  for  review.  These  references  are  listed  in  appendix  A.  Readers  wishing  to  import 
references  into  their  own  Refworks  or  other  bibliographic  management  software  can  contact  the 
ARL  point  of  contact  (Dr.  Elizabeth  Redden)  or  Dr  Michael  Coovert,  University  of  South  Florida, 
to  request  an  import  file. 

2.3  Cognitive  Theory  Objectives 

One  of  the  objectives  for  the  literature  review  was  to  identify  and  review  articles  regarding  theory 
of  multitasking  and  multiple  resources.  Wicken’s  MRT  served  as  a  foundation  for  the  multimodal 
framework,  and  therefore,  articles  that  discussed  or  tested  MRT  concepts  were  coded  specifically 
for  easy  retrieval.  In  addition,  it  was  critical  to  remain  familiar  with  other  related  cognitive  issues; 
thus,  a  cognitive  subcommittee  of  project  members  was  formed.  The  subcommittee  members 
reviewed  literature  (mentioned  in  previous  section),  which  addressed  cognitive  issues  more 
directly,  and  they  provided  summaries  and  presentations  to  the  other  project  members  during  the 
literature  review.  As  the  review  progressed,  the  committee  noted  that  some  important  issues  were 
not  adequately  addressed  by  Wickens’  MRT  (or  any  other  theory).  Although  an  in-depth  analysis 
and  application  of  these  “deeper  issues”  was  outside  the  scope  of  the  present  project,  several  topics 
were  examined  in  more  detail:  parallel  versus  serial  processing;  individual  differences  such  as 
user  preferences  and  brain  hemisphere  dominance;  automaticity;  and  cross-modal  links.  These 
four  examples  have  considerable  theoretical  potential  to  interact  with  the  relationship  between 
multimodal  information  display  and  user  performance.  Therefore,  they  offer  interesting  directions 
for  future  research,  especially  regarding  their  integration  with  the  predictions  of  Wickens’  MRT. 

2.4  Multimodal  Database 

Although  the  new  article  tracking  sheet  was  useful  for  understanding  the  literature  and  coding 
articles  according  to  the  multimodal  framework,  the  paper  format  was  not  conducive  to  queries 
or  comparison  of  components  across  articles.  Consequently,  an  Access2  database  was  developed 
to  provide  an  electronic  version  of  the  article  tracking  sheets  for  reviewed  articles. 


7  .  . 

"Access  is  a  registered  trademark  of  Microsoft  Corporation. 
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Approximately  450  articles  were  reviewed  and  the  respective  article  tracking  sheets  were  entered 
into  the  database  for  each.  The  multimodal  database  includes  all  components  present  on  the 
article  tracking  sheet.  It  supports  queries,  sorting,  and  fdters  as  well  as  side-by-side  comparison 
of  studies  or  experiments  for  all  multimodal  framework  fields.  Furthermore,  because  of  its 
versatile  features,  the  database  is  essentially  capable  of  answering  multimodal  research 
questions,  such  as 

1 .  What  mode  of  information  display  works  best  for  driving? 

2.  Regarding  alerts  or  interruptions,  are  multimodal  displays  better  than  unimodal  displays? 

3.  Is  there  general  support  for  Wickens’  MRT? 

The  multimodal  database  is  an  Access-based  deliverable  that  can  be  made  available  upon  request 
to  the  ARL  advanced  objective  manager  (Dr  Elizabeth  Redden). 


3.  Implications,  Applications,  and  Future  Directions 


The  primary  purpose  of  this  research  was  to  develop  a  framework  to  enable  classification,  evalua¬ 
tion,  and  comparison  of  multimodal  display  research  based  on  task  demands,  display  characteris¬ 
tics,  research  design,  and  individual  differences.  The  identification  of  guiding  principles  for  the 
design  of  multimodal  information  display  was  the  second  objective  and  is  described  in  a  separate 
report.  The  results  of  this  effort,  the  multimodal  framework  and  article  tracking  sheet,  biblio¬ 
graphic  database,  and  searchable  multimodal  database  make  substantial  and  valuable  contributions 
to  the  accumulation  and  interpretation  of  multimodal  research.  References  collected  in  this  effort 
are  listed  in  appendix  A. 

Possibly  the  most  urgent  future  direction  to  address  is  the  inconsistent  use  of  terms  in  the  multi¬ 
modal  literature.  The  most  obvious  nuisance  involves  the  labeling  of  modalities.  For  instance,  a 
number  of  articles  do  not  consider  a  display  to  have  a  “visual  modality”  if  the  visual  information 
exists  in  the  natural  environment.  Other  research  operates  on  the  premise  that  if  visual  cognitive 
resources  are  used  to  process  information,  then  the  term  “visual  modality”  is  appropriate,  regard¬ 
less  of  whether  the  information  is  displayed  on  a  device  or  in  the  natural  environment.  This  simple 
distinction  presents  a  considerable  obstacle  to  the  formation  of  a  knowledge  base  for  the  effects  of 
multi-modal  or  unimodal  displays  where  the  “visual”  modality  is  involved.  A  number  of  other 
critical  terms  are  also  used  inconsistently  (e.g.,  “multimodal”  and  “augment”),  presumably  because 
of  the  variety  of  research  domains  across  which  multimodal  research  takes  place  (e.g.,  human- 
computer  interaction,  psychology,  engineering).  A  concerted  effort  to  amass  a  glossary  of  multi¬ 
modal  and  display  design  terms  would  offer  a  significant  contribution  toward  improving  the 
consistency  of  future  research. 


8 


Another  crucial  objective  for  multimodal  research  is  to  complement  the  present  qualitative  effort 
with  a  quantitative  examination  of  the  effects  of  display  modality  on  human  perfonnance.  There 
are  a  number  of  important  multimodal  research  questions  worthy  of  meta-analysis,  and  their 
evaluation  would  not  only  offer  empirical  support  for  the  guiding  principles,  but  it  would  also 
reveal  meaningful  directions  for  future  research. 
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